Methods and Apparatus to Process Imagery or Audio Content

ABSTRACT

The presently claimed invention relates generally to digital watermarking, and processing imagery (e.g., including video) or audio content. One claim recites a method including: obtaining first media comprising imagery or audio; obtaining second media comprising imagery or audio; aggregating the first media and the second media to yield a unit; and using a programmed electronic processor, examining the unit for the presence of a digital watermark. Another claim recites a method comprising: upon encountering imagery or audio content, performing a screening operation on the imagery or audio content; based at least in part on the result of the screening operation, determining whether to derive or extract identifying data from the imagery or audio content, said act of determining uses a programmed electronic processor; providing the identifying data for storage in an electronic data record. Still another claim recites a method comprising: using a programmed electronic processor, extracting identifying data from data representing imagery or from data representing audio; automatically dialing a telephone number; providing the identifying data for telephonic communication to a device hosting the telephone number; receiving signals representing a human voice command; and providing signals for telephonic communication to a device hosting the telephone number. Of course, other claims and combinations are provided too.

RELATED APPLICATION DATA

This application is a continuation of U.S. patent application Ser. No.12/198,682, filed Aug. 26, 2008 (US 2009-0052735 A1), which is acontinuation of U.S. patent application Ser. No. 11/552,436, filed Oct.24, 2006 (U.S. Pat. No. 7,418,111), which is a continuation ofapplication Ser. No. 10/869,320, filed Jun. 15, 2004 (U.S. Pat. No.7,130,087). The 10/869,320 application is a continuation of applicationSer. No. 09/975,739, filed Oct. 10, 2001 (U.S. Pat. No. 6,750,985),which is a division of application Ser. No. 09/127,502, filed Jul. 31,1998 (U.S. Pat. No. 6,345,104). Application Ser. No. 09/127,502 claimsthe benefit of provisional application No. 60/082,228, filed Apr. 16,1998 (the specification of which is attached as Appendix A).

Application Ser. No. 09/127,502 is also related to application Ser. No.08/649,419, filed May 16, 1996 (now U.S. Pat. No. 5,862,260).Application Ser. No. 09/127,502 is also related to application Ser. No.08/967,693, filed Nov. 12, 1997 (now U.S. Pat. No. 6,122,392), which isa continuation of application Ser. No. 08/614,521, filed Mar. 15, 1996(now U.S. Pat. No. 5,745,604), which is a continuation of applicationSer. No. 08/215,289, filed Mar. 17, 1994 (now abandoned).

The subject matter of this application is also related to that of thepresent assignee's other issued U.S. Pat. Nos. 5,636,292, 5,710,834,5,721,788, 5,748,763, 5,748,783, 5,768,426, 5,850,481, 5,841,978,5,832,119, 5,822,436, 5,841,886, 5,809,160, 6,122,403 and 6,026,193.

FIELD OF THE INVENTION

The present invention relates to methods and systems for inconspicuouslyembedding binary data in security documents, and associatedmethods/systems for detecting/decoding such data. (“Security document”is used herein to refer to negotiable financial instruments (e.g.banknotes, travelers checks, bearer bonds), passports, visas, otherimmigration documents, stock certificates, postal stamps, lotterytickets, sports/concert tickets, etc.) One application of this theinvention is in discouraging counterfeiting of security documents.Another is in transferring machine-readable information through suchdocuments, without alerting human viewers to the presence of suchinformation.

BACKGROUND AND SUMMARY OF THE INVENTION

Digital watermarking (sometimes termed “data hiding” or “dataembedding”) is a growing field of endeavor, with several differentapproaches. The present assignee's work is reflected in the patents andapplications detailed above, together with laid-open PCT applicationWO97/43736. Other work is illustrated by U.S. Pat. Nos. 5,734,752,5,646,997, 5,659,726, 5,664,018, 5,671,277, 5,687,191, 5,687,236,5,689,587, 5,568,570, 5,572,247, 5,574,962, 5,579,124, 5,581,500,5,613,004, 5,629,770, 5,461,426, 5,743,631, 5,488,664, 5,530,759,5,539,735, 4,943,973, 5,337,361, 5,404,160, 5,404,377, 5,315,098,5,319,735, 5,337,362, 4,972,471, 5,161,210, 5,243,423, 5,091,966,5,113,437, 4,939,515, 5,374,976, 4,855,827, 4,876,617, 4,939,515,4,963,998, 4,969,041, and published foreign applications WO 98/02864, EP822,550, WO 97/39410, WO 96/36163, GB 2,196,167, EP 777,197, EP 736,860,EP 705,025, EP 766,468, EP 782,322, WO 95/20291, WO 96/26494, WO96/36935, WO 96/42151, WO 97/22206, WO 97/26733. Some of the foregoingpatents relate to visible watermarking techniques. Other visiblewatermarking techniques (e.g. data glyphs) are described in U.S. Pat.Nos. 5,706,364, 5,689,620, 5,684,885, 5,680,223, 5,668,636, 5,640,647,5,594,809.

Much of the work in data embedding is not in the patent literature butrather is published in technical articles. In addition to the patenteesof the foregoing patents, some of the other workers in this field (whosewatermark-related writings can by found by an author search in theINSPEC or NEXIS databases, among others) include I. Pitas, Eckhard Koch,Jian Zhao, Norishige Morimoto, Laurence Boney, Kineo Matsui, A. Z.Tirkel, Fred Mintzer, B. Macq, Ahmed H. Tewfik, Frederic Jordan, NaohisaKomatsu, Joseph O'Ruanaidh, Neil Johnson, Ingemar Cox, Minerva Yeung,and Lawrence O'Gorman.

The artisan is assumed to be familiar with the foregoing prior art.

In the following disclosure it should be understood that references towatermarking encompass not only the assignee's watermarking technology,but can likewise be practiced with any other watermarking technology,such as those indicated above.

Watermarking can be applied to myriad forms of information. The presentdisclosure focuses on its applications to security documents. However,it should be recognized that the principles discussed below can also beapplied outside this area.

Most of the prior art in image watermarking has focused on pixelatedimagery (e.g. bit-mapped images, JPEG/MPEG imagery, VGA/SVGA displaydevices, etc.). In most watermarking techniques, the luminance or colorvalues of component pixels are slightly changed to effect subliminalencoding of binary data through the image. (This encoding can be donedirectly in the pixel domain, or after the signal has been processed andrepresented differently—e.g. as DCT or wavelet coefficients, or ascompressed data, etc.)

While pixelated imagery is a relatively recent development, securitydocuments—commonly employing line art—go back centuries. One familiarexample is U.S. paper currency. On the one dollar banknote, for example,line art is used in several different ways. One is to form intricatewebbing patterns (sometimes termed “guilloche patterns”) around themargin of the note (generally comprised of light lines on darkbackground). Another is to form gray scale imagery, such as the portraitof George Washington (generally comprised of dark lines on a lightbackground).

There are two basic ways to simulate grey-scales in security documentline art. One is to change the relative spacings of the lines to effecta lightening or darkening of an image region. FIG. 1A shows such anarrangement; area B looks darker than area A due to the closer spacingsof the component lines. The other technique is to change the widths ofthe component lines—wider lines resulting in darker areas and narrowerlines resulting in lighter areas. FIG. 1B shows such an arrangement.Again, area B looks darker than area A, this time due to the greaterwidths of the component lines. These techniques are often used together.Ultimately, a given region simply has more or less ink.

In my U.S. Pat. No. 5,850,481 I introduced, and in my U.S. Pat. No.6,449,377 I elaborated on, techniques for watermarking line art bymaking slight changes to the widths, or positions, of the componentlines. Such techniques are further expanded in the present disclosure.

In several of my cited applications, I discussed various “calibrationsignals” that can be used to facilitate the decoding of watermark datadespite corruption of the encoded image, such as by scaling or rotation.Common counterfeiting techniques—e.g. color photocopying, orscanning/inkjet printing—often introduce such corruption, whetherdeliberately or accidentally. Accordingly, it is important thatwatermarks embedded in security documents be detectable notwithstandingsuch effects. Calibration signals particularly suited for use withsecurity documents are detailed in this disclosure.

In accordance with embodiments of the present invention, securitydocuments are encoded to convey machine-readable multi-bit binaryinformation (e.g. digital watermarks), usually in a manner not alertinghuman viewers that such information is present. The documents can beprovided with overt or subliminal calibration patterns. When a documentincorporating such a pattern is scanned (e.g. by a photocopier), thepattern facilitates detection of the encoded information notwithstandingpossible scaling or rotation of the scan data. The calibration patterncan serve as a carrier for the watermark information, or the watermarkcan be encoded independently. In one embodiment, the watermark and thecalibration pattern are formed on the document by an intaglio process,with or without ink. A photocopier responsive to such markings can takepredetermined action if reproduction of a security document isattempted. A passport processing station responsive to such markings canuse the decoded binary data to access a database having informationconcerning the passport holder. Some such apparatuses detect both thewatermark data and the presence of a visible structure characteristic ofa security document (e.g., the seal of the issuing central bank).

The foregoing and other features and advantages of the presenttechnology will be more readily apparent from the following detaileddescription, which proceeds with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B show prior art techniques for achieving grey-scaleeffects using line art.

FIG. 2 shows a virtual array of grid points that can be imposed on asecurity document image according to an embodiment of the presentinvention.

FIG. 3 shows a virtual array of regions that can be imposed on asecurity document image according to the FIG. 2 embodiment.

FIG. 4 shows an excerpt of FIG. 3 with a line from a line art imagepassing therethrough.

FIG. 5 shows changes to the width of the line of FIG. 3 to effectwatermark encoding.

FIG. 6 shows changes to the position of the line of FIG. 3 to effectwatermark encoding.

FIGS. 7A and 7B show aspects of watermark and calibration blocksaccording to an embodiment of the invention.

FIG. 8 shows an illustrative reference grey-scale calibration tile.

FIGS. 9A-9C show steps in the design of a weave calibration patternaccording to an embodiment of the invention.

FIG. 10 shows the generation of error data used in designing a weavecalibration pattern according to an embodiment of the invention.

FIG. 11 is a block diagram of a passport processing station according toanother embodiment of the invention.

FIG. 12 is a block diagram of a photocopier according to anotherembodiment of the invention.

FIG. 13 is a flow diagram of a method according to one embodiment of theinvention.

DETAILED DESCRIPTION

By way of introduction, the present specification begins with review oftechniques for embedding watermark data in line art, as disclosed in myU.S. Pat. No. 6,449,377.

Referring to FIG. 2, the earlier-described technique employs a grid 10of imaginary reference points arrayed over a line art image. The spacingbetween points is 250 microns in the illustrated arrangement, butgreater or lesser spacings can of course be used.

Associated with each grid point is a surrounding region 12, shown inFIG. 3. As described below, the luminosity (or reflectance) of each ofthese regions 12 is slightly changed to effect subliminal encoding ofbinary data.

Region 12 can take various shapes; the illustrated rounded-rectangularshape is representative only. (The illustrated shape has the advantageof encompassing a fairly large area while introducing fewer visualartifacts than, e.g., square regions.) In other embodiments, squares,rectangles, circles, ellipses, etc., can alternatively be employed.

FIG. 4 is a magnified view of an excerpt of FIG. 3, showing a line 14passing through the grid of points. The width of the line, of course,depends on the particular image of which it is a part. The illustratedline is about 40 microns in width; greater or lesser widths cannaturally be used.

In one encoding technique, shown in FIG. 5, the width of the line iscontrollably varied so as to change the luminosity of the regionsthrough which it passes. To increase the luminosity (or reflectance),the line is made narrower (i.e. less ink in the region). To decrease theluminosity, the line is made wider (i.e. more ink).

Whether the luminance in a given region should be increased or decreaseddepends on the particular watermarking algorithm used. Any algorithm canbe used, by changing the luminosity of regions 12 as the algorithm wouldotherwise change the luminance or colors of pixels in a pixelated image.(Some watermarking algorithms effect their changes in a transformeddomain, such as DCT, wavelet, or Fourier. However, such changes areultimately manifested as changes in luminance or color.)

In an exemplary algorithm, the binary data is represented as a sequenceof −1s and 1s, instead of 0s and 1s. (The binary data can comprise asingle datum, but more typically comprises several. In an illustrativeembodiment, the data comprises 128 bits, some of which areerror-correcting or -detecting bits.)

Each element of the binary data sequence is then multiplied by acorresponding element of a pseudo-random number sequence, comprised of−1s and 1s, to yield an intermediate data signal. Each element of thisintermediate data signal is mapped to a corresponding sub-part of theimage, such as a region 12. (Commonly, each element is mapped to severalsuch sub-parts.) The image in (and optionally around) this region isanalyzed to determine its relative capability to conceal embedded data,and a corresponding scale factor is produced. Exemplary scale factorsmay range from 0 to 3. The scale factor for the region is thenmultiplied by the element of the intermediate data signal mapped to theregion in order to yield a “tweak” or “bias” value for the region. Inthe illustrated case, the resulting tweaks can range from −3 to 3. Theluminosity of the region is then adjusted in accordance with the tweakvalue. A tweak value of −3 may correspond to a −5% change in luminosity;−2 may correspond to −2% change; −1 may correspond to −1% change; 0 maycorrespond to no change; 1 may correspond to +1% change; 2 maycorrespond to +2% change, and 3 may correspond to +5% change. (Thisexample follows the basic techniques described in the Real Time Encoderembodiment disclosed in U.S. Pat. No. 5,710,834.)

In FIG. 5, the watermarking algorithm determined that the luminance ofregion A should be reduced by a certain percentage, while the luminanceof regions C and D should be increased by certain percentages.

In region A, the luminance is reduced by increasing the line width. Inregion D, the luminance is increased by reducing the line width;similarly in region C (but to a lesser extent).

No line passes through region B, so there is no opportunity to changethe region's luminance. This is not fatal to the method, however, sincethe exemplary watermarking algorithm redundantly encodes each bit ofdata in sub-parts spaced throughout the line art image.

The changes to line widths in regions A and D of FIG. 5 are exaggeratedfor purposes of illustration. While the illustrated variance ispossible, most implementations will typically modulate the line width3-50% (increase or decrease).

(Many watermarking algorithms routinely operate within a signal marginof about +/−1% changes in luminosity to effect encoding. That is, the“noise” added by the encoding amounts to just 1% or so of the underlyingsignal. Lines typically don't occupy the full area of a region, so a 10%change to line width may only effect a 1% change to region luminosity,etc. Security documents are different from photographs in that theartwork generally need not convey photorealism. Thus, security documentscan be encoded with higher energy than is used in watermarkingphotographs, provided the result is still aesthetically satisfactory. Toillustrate, localized luminance changes on the order of 10% are possiblein security documents, while such a level of watermark energy inphotographs would generally be considered unacceptable. In somecontexts, localized luminance changes of 20, 30, 50 or even 100% areacceptable.)

In the illustrated technique, the change to line width is a functionsolely of the watermark tweak (or watermark/calibration pattern tweak,as discussed below) to be applied to a single region. Thus, if a linepasses through any part of a region to which a tweak of 2% is to beapplied, the line width in that region is changed to effect the 2%luminance difference. In variant techniques, the change in line width isa function of the line's position in the region. In particular, thechange in line width is a function of the distance between the region'scenter grid point and the line's closest approach to that point. If theline passes through the grid point, the full 2% change is effected. Atsuccessively greater distances, successively smaller changes areapplied. The manner in which the magnitude of the tweak changes as afunction of line position within the region can be determined byapplying one of various interpolation algorithms, such as the bi-linear,bi-cubic, cubic splines, custom curve, etc.

In other variant techniques, the change in line width in a given regionis a weighted function of the tweaks for adjoining or surroundingregions. Thus, the line width in one region may be increased ordecreased in accordance with a tweak value corresponding to one or moreadjoining regions.

Combinations of the foregoing techniques can also be employed.

In the foregoing techniques, it is sometimes necessary to trade-off thetweak values of adjoining regions. For example, a line may pass along aborder between regions, or pass through the point equidistant from fourgrid points (“equidistant zones”). In such cases, the line may besubject to conflicting tweak values—one region may want to increase theline width, while another may want to decrease the line width. (Or bothmay want to increase the line width, but differing amounts.) Similarlyin cases where the line does not pass through an equidistant zone, butthe change in line width is a function of a neighborhood of regionswhose tweaks are of different values. Again, known interpolationfunctions can be employed to determine the weight to be given the tweakfrom each region in determining what change is to be made to the linewidth in any given region.

In the exemplary watermarking algorithm, the average change inluminosity across the security document image is zero, so no generalizedlightening or darkening of the image is apparent. The localized changesin luminosity are so minute in magnitude, and localized in position,that they are essentially invisible (e.g. inconspicuous/subliminal) tohuman viewers.

An alternative technique is shown in FIG. 6, in which line position ischanged rather than line width.

In FIG. 6 the original position of the line is shown in dashed form, andthe changed position of the line is shown in solid form. To decrease aregion's luminosity, the line is moved slightly closer to the center ofthe grid point; to increase a region's luminosity, the line is movedslightly away. Thus, in region A, the line is moved towards the centergrid point, while in region D it is moved away.

It will be noted that the line on the left edge of region A does notreturn to its nominal (dashed) position as it exits the region. This isbecause the region to the left of region A also is to have decreasedluminosity. Where possible, it is generally preferable not to return aline to its nominal position, but instead to permit shifted lines toremain shifted as they enter adjoining regions. So doing permits agreater net line movement within a region, increasing the embeddedsignal level.

Again, the line shifts in FIG. 6 are somewhat exaggerated. More typicalline shifts are on the order of 3-50 microns.

One way to think of the FIG. 6 technique is to employ a magnetismanalogy. The grid point in the center of each region can be thought ofas a magnet. It either attracts or repels lines. A tweak value of −3,for example, may correspond to a strong-valued attraction force; a tweakvalue of +2 may correspond to a middle-valued repulsion force, etc. InFIG. 6, the grid point in region A exhibits an attraction force (i.e. anegative tweak value), and the grid point in region D exhibits arepulsion force (e.g. a positive tweak value).

The magnetic analogy is useful because the magnetic effect exerted on aline depends on the distance between the line and the grid point. Thus,a line passing near a grid point is shifted more in position than a linenear the periphery of the region.

(Actually, the magnetism analogy can serve as more than a conceptualtool. Instead, magnetic effects can be modeled in a computer program andserve to synthesize a desired placement of the lines relative to thegrid points. Arbitrarily customized magnetic fields can be used.)

Each of the variants applicable to FIG. 5 is likewise applicable to FIG.6.

Combinations of the embodiments of FIGS. 5 and 6 can of course be used,resulting in increased watermark energy, better signal-to-noise ratioand, in many cases, less noticeable changes.

In still a further technique, the luminance in each region is changedwhile leaving the line unchanged. This can be effected by sprinklingtiny dots of ink in the otherwise-vacant parts of the region. In highquality printing, of the type used with security documents, droplets onthe order of 3 microns in diameter can be deposited. (Still largerdroplets are still beyond the perception threshold for most viewers.)Speckling a region with such droplets (either in a regular array, orrandom, or according to a desired profile such as Gaussian), can readilyeffect a 1% or so change in luminosity. (Usually dark droplets are addedto a region, effecting a decrease in luminosity. Increases in luminositycan be effected by speckling with a light colored ink, or by forminglight voids in line art otherwise present in a region.) (Actually,production realities often mean that many such microdots will not print,but statistically some will.)

In a variant of the speckling technique, very thin mesh lines can beinserted in the artwork—again to slightly change the luminance of one ormore regions (so-called “background tinting”).

The following portion of the specification reviews a calibration, orsynchronization pattern used in an illustrative security document tofacilitate proper registration of the watermark data for decoding. Itmay be helpful to begin by reviewing further details about theillustrative watermarking method.

Referring to FIG. 7A, an exemplary watermark is divided into “cells”that are 250 microns on a side, each conveying a single bit ofinformation. The cells are grouped into a “block” having 128 cells on aside (i.e. 16,384 cells per block). The blocks are tiled across theregion being watermarked (e.g. across the face of a security document).

As noted, the watermark payload consists of 128 bits of data. Each bitis represented by 128 different cells within each block. (The mapping ofbits to cells can be pseudo-random, sequential, or otherwise.) The128“0”s and “1”s of the watermark data are randomized into substantiallyequal-probability “1”s and “−1”s by a pseudo-random function to reducewatermark visibility. Where a cell has a value of “1,” the luminance ofthe corresponding area of the image is slightly increased; where a cellhas a value of “−1,” the luminance of the corresponding area of theimage is slightly decreased (or vice versa). In some embodiments, thelocalized changes to image luminance due to the +1/−1 watermark cellvalues are scaled in accordance with data-hiding attributes of the localarea (e.g. to a range of +/−4 digital numbers) to increase therobustness of the watermark without compromising its imperceptibility.

It should be noted that a single watermark “cell” commonly encompasses alarge number of ink droplets. In high resolution printing, as iscommonly used in security documents (e.g. 5000 microdroplets per inch),a single watermark cell may encompass a region of 50 droplets by 50droplets. In other embodiments, a cell may encompass greater or lessernumbers of droplets.

Decoding a watermark requires precise re-registration of the scanneddocument image, so the watermark cells are located where expected. Tofacilitate such registration, a calibration signal can be employed.

An exemplary calibration signal is a geometrical pattern having a knownFourier-Mellin transform. As described in U.S. Pat. No. 5,862,260, whena known pattern is transformed into the Fourier domain, and then furthertransformed into the Fourier-Mellin domain, the transformed dataindicates the scale and rotation of the pattern. If this pattern isreplicated on a security document that is thereafter scanned (as noted,scanning commonly introduces rotation, and sometimes scaling), the F-Mtransform data indicates the scale and rotation of the scanned data,facilitating virtual re-registration of the security document image forwatermark detection.

As shown in FIG. 7B, an illustrative geometrical calibration pattern isa block, 3.2 cm on a side. The block comprises a 16×16 array ofsubstantially identical tiles, each 2 mm on a side. Each tile, in term,comprises an 8×8 array of component cells.

As described below, the geometrical calibration pattern in theillustrated embodiment is a visible design feature on the securitydocument. Accordingly, unlike the watermark data, the calibrationpattern does not have to be limited to a small range of digital numbersin order to keep it substantially hidden among other features of thedocument. Also unlike the watermark data, the illustrated calibrationpattern is not locally scaled in accordance with data hiding attributesof the security document image.

It is possible to print rectangular grids of grey-scaled ink on adocument to serve as a calibration pattern. However, aestheticconsiderations usually discourage doing so. Preferable is to realize thecalibration pattern in a more traditional art form, such as a seeminglyrandom series of intertwining lines, forming a weave-like pattern thatis printed across part or all of the document.

To create this weave-like calibration pattern, a designer first definesan 8×8 cell reference calibration tile. Each cell in the tile isassigned a grey-scale value. In the illustrated embodiment, valueswithin 2-10 percent of each other are used, although this is notessential. An exemplary reference calibration tile is shown in FIG. 8(assuming 8-bit quantization).

The Fourier-Mellin transform of a block derived from this referencecalibration tile will serve as the key by which the scale and rotationof a scanned security document image are determined.

There is some optimization that may be done in selecting/designing thepattern of grey-scale values that define the reference calibration tile.The pattern should have a F-M transform that is readily distinguishedfrom those of other design and watermark elements on the securitydocument. One design procedure effects a trial F-M transform of the restof the security document design, and works backwards from this data toselect a reference calibration tile that is readily distinguishable.

Once a reference tile pattern is selected, the next steps iterativelydefine a tile having a weave-like pattern whose local luminance valuesapproximately match the reference tile's grey-scale pattern.

Referring to FIG. 9A, the first such step is to select points on thebottom and left side edges of the tile where lines are to cross the tileboundaries. The angles at which the lines cross these boundaries arealso selected. (In the illustrated embodiment, these points and anglesare selected arbitrarily, although in other embodiments, the choices canbe made in conformance with an optimizing design procedure.)

The selected points and angles are then replicated on the correspondingright and top edges of the tile. By this arrangement, lines exiting thetop of one tile seamlessly enter the bottom of the adjoining tile at thesame angle. Likewise, lines exiting either side of a tile seamlesslyjoin with lines in the laterally adjoining blocks.

The designer next establishes trial line paths snaking through the tile(FIGS. 9B, 9C), linking arbitrarily matched pairs of points on thetile's edges. (These snaking paths are sometimes termed “worms.”)Desirably, these paths pass through each of the 64 component cellsforming the tile, with the total path length through each cell beingwithin +/−30% of the average path length through all cells. (This trialrouting can be performed with pencil and paper, but more commonly isdone on a computer graphics station, with a mouse, light pen, or otherinput device being manipulated by the designer to establish therouting.) In the illustrated embodiment, the lines have a width of about30-100 microns, and an average spacing between lines of about 100-400microns, although these parameters are not critical.

Turning next to FIG. 10, the trial tile is assembled with like tiles toform a 16×16 trial block (3.2 cm on a side), with a repetitive weavepattern formed by replication of the line pattern defined on the 8×8cell trial tile. This trial block is then converted into grey-scalevalues. The conversion can be done by scanning a printed representationof the trial block, or by computer analysis of the line lengths andpositions. The output is a 128×128 array of grey-scale values, eachvalue corresponding to the luminance of a 250 micron cell within thetrial block.

This grey-scale data is compared with grey-scale data provided byassembling 256 of the reference calibration tiles (each an 8×8 array ofcells) into a 16×16 calibration pattern block. In particular, thegrey-scale array resulting from the trial block is subtracted from thegrey-scale array resulting from the reference block, generating a128×128 array of error values. This error data is used to tweak thearrangement of lines in the trial block.

In cells of the trial calibration block where the error value ispositive, the line is too long. That is, the pattern is too dark inthose cells (i.e. it has a low luminance grey-scale value), due to asurplus of line length (i.e. too much ink). By shortening the linelength in those cells, their luminance is increased (i.e. the cell islightened). Shortening can be effected by straightening curved arcs, orby relocating a line's entrance and exit points in a cell so lessdistance is traversed through the cell.

Conversely, in cells where the error value is negative, the line is tooshort. By increasing the line length in such cells, their luminance isdecreased (i.e. the cell is darkened). Increasing the line lengththrough a cell can be accomplished by increasing the curvature of theline in the cell, or by relocating a line's entrance and exit pointsalong the boundary of the cell, so more distance is traversed throughthe cell.

A computer program is desirably employed to effect the foregoing changesin line routing to achieve the desired darkening or lightening of eachcell.

After line positions in the trial calibration block have been tweaked inthis fashion, the trial block is again converted to grey-scale values,and again subtracted from the reference block. Again, an array of errorvalues is produced. The positions of the lines are then further tweakedin accordance with the error values.

The foregoing steps of tweaking line routes in accordance with errorsignals, converting anew into grey-scale, and computing new errorvalues, is repeated until the luminance of the resulting weave patternin the trial block is arbitrarily close to the luminance of thereference block. Four of five iterations of this procedure commonlysuffice to converge on a final calibration block.

(It will be noted that the initial tile pattern created by the designeris done at the tile level—8×8 cells. After the initial trial tile iscreated, subsequent processing proceeds at the block level (128×128cells). A common result of the iterative design procedure is that thecomponent tiles lose their uniformity. That is, the pattern of lines ina tile at a corner of the final calibration block will generally beslightly different than the pattern of lines in a tile near the centerof the block.)

After the final calibration block pattern has been established as above,the blocks are tiled repetitively over some or all of the securitydocument, and can serve either as a background design element, or as amore apparent element of the design. By printing this weave pattern inan ink color close to the paper substrate color, the patterning ishighly unobtrusive. (If a highly contrasting ink color is used, and ifthe pattern extends over most or all of the security document, it may bedesirable to employ a brighter luminance paper than otherwise, since theweave pattern effectively darkens the substrate.)

As noted in my U.S. Pat. No. 5,862,260, the Fourier-Mellin transform hasthe property that the same output pattern is produced, regardless ofrotation or scaling of the input image. The invariant output pattern isshifted in one dimension proportional to image rotation, and shifted inanother dimension proportional to image scaling. When an image whose F-Mtransform is known, is thereafter rotated and/or scaled, the degree ofrotation and scaling can be determined by observing the degree of shiftof the transformed F-M pattern in the two dimensions. Once the rotationand scale are known, reciprocal processing of the image can be performedto restore the image to its original orientation and scale.

In the above-described embodiment, the calibration block pattern has aknown F-M transform. When a security document incorporating such apattern is scanned (e.g. by a photocopier, a flatbed scanner, afacsimile machine, etc.), the resulting data can be F-M transformed. Theknown F-M pattern is then identified in the transformed data, and itstwo-dimensional shift indicates the scale and rotation corruption of thescanned security document data. With these parameters known,misregistration of the security document—including scale and rotationcorruption—can be backed-off, and the security document data restored toproper alignment and scale. In this re-registered state, the watermarkcan be detected. (In alternative embodiments, the original scan data isnot processed to remove the scale/rotation effects. Instead, subsequentprocessing proceeds with the data in its corrupted state, and takes intoaccount the specific corruption factor(s) to nonetheless yield accuratedecoding, etc.)

The just-described calibration pattern and design procedure, of course,are just exemplary, and are subject to numerous modifications. Thedimensions can be varied at will. It is not essential that the cell sizeof the calibration tiles match that of the watermark. Nor do the cellssizes need to be integrally related to each other. Nor does thecalibration pattern need to be implemented as lines; other ink patternscan alternatively be used to approximate the grey-scale referencepattern

There is no requirement that the lines snake continuously through thetiles. A line can connect to just a single edge point of a tile,resulting in a line that crosses that tile boundary, but no other. Or aline can both begin and end in a single tile, and not connect to anyother.

While darker lines on a lighter background are illustrated, lighterlines on a darker background can alternatively be employed.

The iterative design procedure can employ the F-M transform (or othertransform). For example, the trial block pattern can be transformed tothe F-M domain, and there compared with the F-M transform of thereference block. An F-M domain error signal can thus be obtained, andthe routing of the lines can be changed in accordance therewith.

Although the illustrated embodiment tweaked the cell-based grey-scalesof the calibration block by changing line curvature and position, otherluminance changing techniques can be employed. For example, the width ofthe weave lines can be locally changed, or small ink dots can beintroduced into certain cell areas.

The foregoing (and following) discussions contemplate that the watermarkand/or calibration pattern is printed at the same time as (indeed,sometimes as part of) the line art on the security document. In manyapplications it is desirable to provide the calibration pattern on thesecurity document substrate prior to printing. The markings can be inkapplied by the manufacturer, or can be embossings applied, e.g., byrollers in the paper-making process. (Such textural marking is discussedfurther below.) Or, the markings can be applied by the security documentprinter, as a preliminary printing operation, such as by offsetprinting. By using an ink color/density that is already closely matchedto the underlying tint of the paper stock, the manufacturer of the papercan introduce less tinting during its manufacture. Such tinting willeffectively be replaced by the preliminary printing of thewatermark/calibration pattern on the blank paper.

Calibration signals entirely different than those detailed above canalso be used. Calibration signals that are optimized to detect rotation,but not scaling, can be employed when scaling is not a serious concern.DCT and Fourier transforms provide data that is readily analyzed todetermine rotation. A calibration signal can be tailored to stand out ina typically low-energy portion of the transformed spectrum (e.g. aseries of fine lines at an inclined angle transforms to a usually vacantregion in DCT space), and the scanned image can be transformed to theDCT/Fourier domains to examine any shift in the calibration signal (e.g.a shift in the spatial frequency representation of the inclined lines).

In some security documents, the just-described calibration weave isprinted independently of the watermark encoding. In other embodiments,the weave serves as the lines whose widths, locations, etc., aremodulated by the watermark data, as detailed herein and in U.S. Pat. No.6,449,377.

In an illustrative embodiment, the printing of the security document isachieved by intaglio printing. Intaglio is a well known printing processemploying a metal plate into which the security document pattern isetched or engraved. Ink is applied to the plate, filling the etchedrecesses/grooves. Paper is then pressed into the plate at a very highpressure (e.g. 10-20 tons), both raised-inking and slightly deforming(texturing) the paper.

Although ink is commonly used in the intaglio process, it need not be incertain embodiments of the present invention. Instead, the papertexturing provided by the intaglio pressing—alone—can suffice to conveywatermark data. (Texturing of a medium to convey watermark informationis disclosed in various of my prior applications, including U.S. Pat.No. 5,850,481.)

To illustrate, an intaglio plate was engraved (using a numericallycontrolled engraving apparatus), to a depth of slightly less than 1 mm,in accordance with a 3.2×3.2 cm. noise-like block of watermark data. Thewatermark data was generated as described above (e.g. 128 bits of data,randomly distributed in a 128×128 cell array), and summed with acorrespondingly-sized block of calibration data (implemented as discretegrey-scaled cells, rather than the line/weave pattern detailed above).In this embodiment, the data was not kept within a small range ofdigital numbers, but instead was railed to a full 8-bit dynamic range.)

This textured paper was placed—textured extrema down—on the platen of anconventional flatbed scanner (of the sort commonly sold as an accessoryfor personal computers), and scanned. The resulting image data was inputto Adobe's Photoshop image processing software, version 4.0, whichincludes Digimarc watermark reader software. The software readilydetected the watermark from the textured paper, even when the paper wasskewed on the scanner platen.

The optical detection process by which a seemingly blank piece of papercan reliably convey 128 bits of data through an inexpensive scanner hasnot been analyzed in detail; the degree of localized reflection from thepaper may be a function of whether the illuminated region is concave orconvex in shape. Regardless of the explanation, it is a remarkablephenomenon to witness.

Experiments have also been conducted using traditional opaque inks.Again, the watermark can reliably be read.

In addition to the just-described technique for “reading” intagliomarkings by a conventional scanner, a variant technique is disclosed inVan Renesse, Optical Inspection Techniques for Security Instrumentation,SPIE Proc. Vol. 2659, pp. 159-167 (1996), and can alternatively be usedin embodiments according to the present invention.

Although intaglio is a preferred technique for printing securitydocuments, it is not the only such technique. Other familiar techniquesby which watermarks and calibration patterns can be printed includeoffset litho and letterpress, as well as inkjet printing, xerographicprinting, etc. And, as noted, textured watermarking can be effected aspart of the paper-making process, e.g. by high pressure texturedrollers.

In still other embodiments, the watermark and/or calibration(“information”) patterns are not printed on the security documentsubstrate, but rather are formed on or in an auxiliary layer that islaminated with a base substrate. If a generally clear laminate is used,the information patterns can be realized with opaque inks, supplementingthe design on the underlying substrate. Or the added information can beencoded in textural form. Combinations of the foregoing can similarly beused.

To retrofit existing security document designs with informationpatterns, the existing artwork must be modified to effect the necessaryadditions and/or tweaks to localized security document luminance and/ortexture.

When designing new security documents, it would be advantageous tofacilitate integration of information patterns into the basic design.One such arrangement is detailed in the following discussion.

Many security documents are still designed largely by hand. A designerworks at a drafting table or computer workstation, and spends many hourslaying-out minute (e.g. 5 mm×5 mm) excerpts of the design. To aidintegration of watermark and/or calibration pattern data in thisprocess, an accessory layout grid can be provided, identifying thewatermark “bias” (e.g. −3 to +3) that is to be included in each 250micron cell of the security document. If the accessory grid indicatesthat the luminance should be slightly increased in a cell (e.g. 1%), thedesigner can take this bias in mind when defining the composition of thecell and include a touch less ink than might otherwise be included.Similarly, if the accessory grid indicates that the luminance should besomewhat strongly increased in a cell (e.g. 5%), the designer can againbear this in mind and try to include more ink than might otherwise beincluded. Due to the substantial redundancy of most watermark encodingtechniques, strict compliance by the designer to these guidelines is notrequired. Even loose compliance can result in artwork that requireslittle, if any, further modification to reliably convey watermark and/orcalibration information.

Such “designing-in” of embedded information in security documents isfacilitated by the number of arbitrary design choices made by securitydocument designers. A few examples from U.S. banknotes include the curlsin the presidents' hair, the drape of clothing, the clouds in the skies,the shrubbery in the landscaping, the bricks in the pyramid, the fillpatterns in the lettering, and the great number of arbitrary guillochepatterns and other fanciful designs, etc. All include curves, folds,wrinkles, shadow effects, etc., about which the designer has widediscretion in selecting local luminance, etc. Instead of making suchchoices arbitrarily, the designer can make these choices deliberately soas to serve an informational—as well as an aesthetic—function.

To further aid the security document designer, data defining severaldifferent information-carrying patterns (both watermark and/orcalibration pattern) can be stored on mass storage of a computerworkstation and serve as a library of design elements for futuredesigns. The same user-interface techniques that are employed to pickcolors in image-editing software (e.g. Adobe Photoshop) and filltextures in presentation programs (e.g. Microsoft PowerPoint) cansimilarly be used to present a palette of information patterns to asecurity document designer. Clicking on a visual representation of thedesired pattern makes the pattern available for inclusion in a securitydocument being designed (e.g. filling a desired area).

In the embodiment earlier-described, the calibration pattern is printedas a visible artistic element of the security document. However, thesame calibration effect can be provided subliminally if desired. Thatis, instead of generating artwork mimicking the grey-scale pattern ofthe reference calibration block, the reference calibration block canitself be encoded into the security document as small changes in localluminance. In many such embodiments, the bias to localized documentluminance due to the calibration pattern is simply added to the bias dueto the watermark data, and encoded like the watermark data (e.g. aslocalized changes to the width or position of component line-art lines,as inserted ink droplets, etc.).

The uses to which the 128 bits of watermark data can be put in securitydocuments are myriad. Many are detailed in the materials cited above.Examples include postal stamps encoded with their value, or with the zipcode of the destination to which they are addressed (or from which theywere sent); banknotes encoded with their denomination, and their dateand place of issuance; identification documents encoded withauthentication information by which a person's identify can be verified;etc., etc.

The encoded data can be in a raw form—available to any reader having therequisite key data (in watermarking techniques where a key data isused), or can be encrypted, such as with public key encryptiontechniques, etc. The encoded data can embody information directly, orcan be a pointer or an index to a further collection of data in whichthe ultimate information desired is stored.

For example, watermark data in a passport need not encode a completedossier of information on the passport owner. Instead, the encoded datacan include key data (e.g. a social security number) identifying aparticular record in a remote database in which biographical datapertaining to the passport owner is stored. A passport processingstation employing such an arrangement is shown in FIG. 11.

To decode watermark data, the security document must be converted intoelectronic image data for analysis. This conversion is typicallyperformed by a scanner.

Scanners are well known, so a detailed description is not provided here.Suffice it to say that scanners conventionally employ a line of closelyspaced photodetector cells that produce signals related to the amount ofthe light reflected from successive swaths of the document. Mostinexpensive consumer scanners have a resolution of 300 dots per inch(dpi), or a center to center spacing of component photodetectors ofabout 84 microns. Higher quality scanners of the sort found in mostprofessional imaging equipment and photocopiers have resolutions of 600dpi (42 microns), 1200 dpi (21 microns), or better.

Taking the example of a 300 dpi scanner (84 micron photodetectorspacing), each 250 micron region 12 on the security document willcorrespond to about a 3×3 array of photodetector samples. Naturally,only in rare instances will a given region be physically registered withthe scanner so that nine photodetector samples capture the luminance inthat region, and nothing else. More commonly, the image is rotated withrespect to the scanner photodetectors, or is longitudinally misaligned(i.e. some photodetectors image sub-parts of two adjoining regions).However, since the scanner oversamples the regions, the luminance ofeach region can unambiguously be determined.

In one embodiment, the scanned data from the document is collected in atwo dimensional array of data and processed to detect the embeddedcalibration information. The scanner data is then processed to effect avirtual re-registration of the document image. A software program nextanalyzes the statistics of the re-registered data (using the techniquesdisclosed in my prior writings) to extract the bits of the embeddeddata.

(Again, the reference to my earlier watermark decoding techniques isexemplary only. Once scanning begins and the data is available insampled form, it is straightforward to apply any other watermarkdecoding technique to extract a correspondingly-encoded watermark. Someof these other techniques employ domain transformations (e.g. towavelet, DCT, or Fourier domains, as part of the decoding process).)

In a variant embodiment, the scanned data is not assembled in a completearray prior to processing. Instead, it is processed in real-time, as itis generated, in order to detect embedded watermark data without delay.(Depending on the parameters of the scanner, it may be necessary to scana half-inch or so of the document before the statistics of the resultingdata unambiguously indicate the presence of a watermark.)

In other embodiments, hardware devices are provided with the capabilityto recognize embedded watermark data in any document images theyprocess, and to respond accordingly.

One example is a color photocopier. Such devices employ a color scannerto generate sampled (pixel) data corresponding to an input media (e.g. adollar bill). If watermark data associated with a security document isdetected, the photocopier can take one or more steps.

One option is simply to interrupt copying, and display a messagereminding the operator that it is illegal to reproduce currency.

Another option is to dial a remote service and report the attemptedbanknote reproduction. Photocopiers with dial-out capabilities are knownin the art (e.g. U.S. Pat. No. 5,305,199) and are readily adapted tothis purpose. The remote service can be an independent service, or canbe a government agency.

Yet another option is to permit the copying, but to insert forensictracer data in the resultant copy. This tracer data can take variousforms. Steganographically encoded binary data is one example. An exampleis shown in U.S. Pat. No. 5,568,268. The tracer data can memorialize theserial number of the machine that made the copy and/or the date and timethe copy was made. To address privacy concerns, such tracer data is notnormally inserted in all photocopied output, but is inserted only whenthe subject being photocopied is detected as being a security document.(An example of such an arrangement is shown in FIG. 12.)

Desirably, the scan data is analyzed on a line-by-line basis in order toidentify illicit photocopying with a minimum of delay. If a securitydocument is scanned, one or more lines of scanner output data may beprovided to the photocopier's reprographic unit before the recognitiondecision has been made. In this case the photocopy will have tworegions: a first region that is not tracer-marked, and a second,subsequent region in which the tracer data has been inserted.

Photocopiers with other means to detect not-to-be-copied documents areknown in the art, and employ various response strategies. Examples aredetailed in U.S. Pat. Nos. 5,583,614, 4,723,149, 5,633,952, 5,640,467,and 5,424,807.

Another hardware device that can employ the foregoing principles is astandalone scanner. A programmed processor (or dedicated hardware)inside the scanner analyzes the data being generated by the device, andresponds accordingly.

Yet another hardware device that can employ the foregoing principles isa printer. A processor inside the device analyzes graphical image datato be printed, looking for watermarks associated with securitydocuments.

For both the scanner and printer devices, response strategies caninclude disabling operation, or inserting tracer information. (Suchdevices typically do not have dial-out capabilities.)

Again, it is desirable to process the scanner or printer data as itbecomes available, so as to detect any security document processing witha minimum of delay. Again, there will be some lag time before adetection decision is made. Accordingly, the scanner or printer outputwill be comprised of two parts, one without the tracer data, and anotherwith the tracer data.

Many security documents already include visible structures that can beused as aids in banknote detection (e.g. the seal of the issuer, andvarious geometrical markings on U.S. currency). In accordance with afurther aspect of the present invention, a security document is analyzedby an integrated system that considers both the visible structures andwatermark-embedded data.

Visible security document structures can be sensed using known patternrecognition techniques. Examples of such techniques are disclosed inU.S. Pat. Nos. 5,321,773, 5,390,259, 5,533,144, 5,539,841, 5,583,614,5,633,952, 4,723,149, 5,692,073, and 5,424,807 and laid-open foreignapplications EP 649,114 and EP 766,449.

In photocopiers (and the like) equipped to detect both visiblestructures and watermarks from security documents, the detection ofeither can cause one or more of the above-noted responses to beinitiated (FIG. 12).

Again, scanners and printers can be equipped with a similarcapability—analyzing the data for either of these security documenthallmarks. If either is detected, the software (or hardware) respondsaccordingly.

Identification of security documents by watermark data provides animportant advantage over recognition by visible structures—it cannot soeasily be defeated. A security document can be doctored (e.g. bywhite-out, scissors, or less crude techniques) to remove/obliterate thevisible structures. Such a document can then be freely copied on eithera visible structure-sensing photocopier or scanner/printer installation.The removed visible structure can then be added back in via a secondprinting/photocopying operation. If the printer is not equipped withsecurity document-disabling capabilities, image-editing tools can beused to insert visible structures back into image data sets scanned fromsuch doctored documents, and the complete document can then be freelyprinted. By additionally including embedded watermark data in thesecurity document, and sensing same, such ruses will not succeed.

(A similar ruse is to scan a security document image on a non-securitydocument-sensing scanner. The resulting image set can then be edited byconventional image editing tools to remove/obliterate the visiblestructures. Such a data set can then be printed—even on aprinter/photocopier that examines such data for the presence of visiblestructures. Again, the missing visible structures can be inserted by asubsequent printing/photocopying operation.)

Desirably, the visible structure detector and the watermark detector areintegrated together as a single hardware and/or software tool. Thisarrangement provides various economies, e.g., in interfacing with thescanner, manipulating pixel data sets for pattern recognition andwatermark extraction, electronically re-registering the image tofacilitate pattern recognition/watermark extraction, issuing controlsignals (e.g. disabling) signals to the photocopier/scanner, etc.

While the foregoing apparatuses are particularly concerned withcounterfeit deterrence, the embedded markings can also serve otherfunctions. Examples include banknote processing machines that performdenomination sorting, counterfeit detection, and circulation analysisfunctions. (I.e., banknotes with certain markings may be distributedthrough known sources, and their circulation/distribution cansubsequently be monitored to assist in macro-economic analyses.)

From the foregoing, it will be recognized that various embodimentsaccording to the present invention provide techniques for embeddingmulti-bit binary data in security documents, and provide for thereliable extraction of such data even in the presence of various formsof corruption (e.g. scale and rotation).

(To provide a comprehensive disclosure without unduly lengthening thefollowing specification, applicant incorporates by reference the patentsand applications cited above.)

Having described and illustrated the principles of my invention withreference to several illustrative embodiments, it will be recognizedthat these embodiments are exemplary only and should not be taken aslimiting the scope of my invention. Guided by the foregoing teachings,it should be apparent that other watermarking, decoding, andanti-counterfeiting technologies can be substituted for, and/or combinedwith, the elements detailed above to yield advantageous effects. Otherfeatures disclosed in my earlier applications can similarly be employedin embodiments of the technology detailed herein. (Thus, I have not herebelabored application of each of the techniques disclosed in my earlierapplications—e.g. use of neural networks for watermark detectors—to thepresent subject matter since same is fairly taught by reading thepresent disclosure in the context of my earlier work.)

While the technology has been described with reference to embodimentsemploying regular rectangular arrays of cells, those skilled in the artwill recognize that other arrays—neither rectangular nor regular—canalternatively be used.

While the embodiments have described the calibration patterns asadjuncts to digital watermarks—facilitating their detection, suchpatterns have utility apart from digital watermarks. One example is inre-registering scanned security document image data to facilitatedetection of visible structures (e.g. detection of the seal of theissuer, using known pattern recognition techniques). Indeed, the use ofsuch calibration patterns to register both watermark and visiblestructure image data for recognition is an important economy that can begained by integration a visible structure detector and a watermarkdetector into a single system.

Although security documents have most commonly been printed on paper(e.g. cotton/linen), other substrates are gaining in popularity (e.g.synthetics, such as polymers) and are well (or better) suited for usewith the above-described techniques.

The embodiments detailed above can be implemented in dedicated hardware(e.g. ASICs), programmable hardware, and/or software.

In view of the many possible embodiments to which the principles of theabove-described technology may be put, it should be recognized that thedetailed embodiments are illustrative only and should not be taken aslimiting the scope of my invention. Rather, I claim as my invention allsuch embodiments as may come within the scope and spirit of thefollowing claims and equivalents thereto.

APPENDIX A Watermarking Methods, Apparatuses, and Applications

(To provide a comprehensive disclosure without unduly lengthening thefollowing specification, applicants incorporate by reference the citedpatent documents.)

Watermarking is a quickly growing field of endeavor, with severaldifferent approaches. The present assignee's work is reflected in U.S.Pat. Nos. 5,710,834, 5,636,292, 5,721,788, allowed U.S. application Ser.Nos. 08/327,426, 08/598,083, 08/436,134 (to issue as U.S. Pat. No.5,748,763), 08/436,102 (to issue as U.S. Pat. No. 5,748,783), and08/614,521 (to issue as U.S. Pat. No. 5,745,604), and laid-open PCTapplication WO97/43736. Other work is illustrated by U.S. Pat. Nos.5,734,752, 5,646,997, 5,659,726, 5,664,018, 5,671,277, 5,687,191,5,687,236, 5,689,587, 5,568,570, 5,572,247, 5,574,962, 5,579,124,5,581,500, 5,613,004, 5,629,770, 5,461,426, 5,743,631, 5,488,664,5,530,759, 5,539,735, 4,943,973, 5,337,361, 5,404,160, 5,404,377,5,315,098, 5,319,735, 5,337,362, 4,972,471, 5,161,210, 5,243,423,5,091,966, 5,113,437, 4,939,515, 5,374,976, 4,855,827, 4,876,617,4,939,515, 4,963,998, 4,969,041, and published foreign applications WO98/02864, EP 822,550, WO 97/39410, WO 96/36163, GB 2,196,167, EP777,197, EP 736,860, EP 705,025, EP 766,468, EP 782,322, WO 95/20291, WO96/26494, WO 96/36935, WO 96/42151, WO 97/22206, WO 97/26733. Some ofthe foregoing patents relate to visible watermarking techniques. Othervisible watermarking techniques (e.g. data glyphs) are described in U.S.Pat. Nos. 5,706,364, 5,689,620, 5,684,885, 5,680,223, 5,668,636,5,640,647, 5,594,809.

Most of the work in watermarking, however, is not in the patentliterature but rather in published research. In addition to thepatentees of the foregoing patents, some of the other workers in thisfield (whose watermark-related writings can by found by an author searchin the INSPEC database) include I. Pitas, Eckhard Koch, Jian Zhao,Norishige Morimoto, Laurence Boney, Kineo Matsui, A. Z. Tirkel, FredMintzer, B. Macq, Ahmed H. Tewfik, Frederic Jordan, Naohisa Komatsu, andLawrence O'Gorman.

The artisan is assumed to be familiar with the foregoing prior art.

In the following disclosure it should be understood that references towatermarking encompass not only the assignee's watermarking technology,but can likewise be practiced with any other watermarking technology,such as those indicated above.

Watermarking can be applied to myriad forms of information. Theseinclude imagery (including video) and audio—whether represented indigital form (e.g. an image comprised of pixels, digital video, etc.),or in an analog representation (e.g. non-sampled music, printed imagery,banknotes, etc.) Watermarking can be applied to digital content (e.g.imagery, audio) either before or after compression. Watermarking canalso be used in various “description” or “synthesis” languagerepresentations of content, such as Structured Audio, Csound, NetSound,SNHC Audio and the like (c.f. http://sound.media.mit.edu/mpeg4/) byspecifying synthesis commands that generate watermark data as well asthe intended audio signal. Watermarking can also be applied to ordinarymedia, whether or not it conveys information. Examples include paper,plastics, laminates, paper/film emulsions, etc. A watermark can embed asingle bit of information, or any number of bits.

The physical manifestation of watermarked information most commonlytakes the form of altered signal values, such as slightly changed pixelvalues, picture luminance, picture colors, DCT coefficients,instantaneous audio amplitudes, etc. However, a watermark can also bemanifested in other ways, such as changes in the surface microtopologyof a medium, localized chemical changes (e.g. in photographicemulsions), localized variations in optical density, localized changesin luminescence, etc. Watermarks can also be optically implemented inholograms and conventional paper watermarks.

One improvement to existing technology is to employ established webcrawler services (e.g. AltaVista, Excite, or Inktomi) to search forwatermarked content (on the Web, in internet news groups, BBS systems,on-line systems, etc.) in addition to their usual datacollecting/indexing operations. Such crawlers can download files thatmay have embedded watermarks (e.g. *.JPG, *.WAV, etc.) for lateranalysis. These files can be processed, as described below, in realtime. More commonly, such files are queued and processed by a computerdistinct from the crawler computer. Instead of performing watermark-readoperations on each such file, a screening technique can be employed toidentify those most likely to be conveying watermark data. One suchtechnique is to perform a DCT operation on an image, and look forspectral coefficients associated with certain watermarking techniques(e.g. coefficients associated with an inclined embedded subliminalgrid). To decode spread-spectrum based watermarks, the analyzingcomputer requires access to the noise signal used to spread the datasignal. In one embodiment, interested parties submit their noise/keysignals to the crawler service so as to enable their marked content tobe located. The crawler service maintains such information inconfidence, and uses different noise signals in decoding an image (imageis used herein as a convenient shorthand for imagery, video, and audio)until watermarked data is found (if present). This allows the use of webcrawlers to locate content with privately-coded watermarks, instead ofjust publicly-coded watermarks as is presently the case. The queueing ofcontent data for analysis provides certain opportunities forcomputational shortcuts. For example, like-sized images (e.g. 256×256pixels) can be tiled into a larger image, and examined as a unit for thepresence of watermark data. If the decoding technique (or the optionalpre-screening technique) employs a DCT transform or the like, the blocksize of the transform can be tailored to correspond to the tile size (orsome integral fraction thereof). Blocks indicated as likely havingwatermarks can then be subjected to a full read operation. If the queueddata is sorted by file name, file size, or checksum, duplicate files canbe identified. Once such duplicates are identified, the analysiscomputer need consider only one instance of the file. If watermark datais decoded from such a file, the content provider can be informed ofeach URL at which copies of the file were found.

Some commentators have observed that web crawler-based searches forwatermarked images can be defeated by breaking a watermarked image intosub-blocks (tiles). HTML instructions, or the like, cause the sub-blocksto be presented in tiled fashion, recreating the complete image.However, due to the small size of the component sub-blocks, watermarkreading is not reliably accomplished.

This attack is overcome by instructing the web-crawler to collect thedisplay instructions (e.g. HTML) by which image files are positioned fordisplay on a web page, in addition to the image files themselves. Beforefiles collected from a web page are scrutinized for watermarks, they canbe concatenated in the arrangement specified by the displayinstructions. By this arrangement, the tiles are reassembled, and thewatermark data can be reliably recovered.

Another such postulated attack against web crawler detection of imagewatermarks is to scramble the image (and thus the watermark) in a file,and employ a Java applet or the like to unscramble the image prior toviewing. Existing web crawlers inspect the file as they find it, so thewatermark is not detected. However, just as the Java descrambling appletcan be invoked when a user wishes access to a file, the same applet cansimilarly be employed in a web crawler to overcome such attemptedcircumvention of watermark detection.

Although “content” can be located and indexed by various web crawlers,the contents of the “content” are unknown. A *.JPG file, for example,may include pornography, a photo of a sunset, etc.

Watermarks can be used to indelibly associate meta-data within content(as opposed to stored in a data structure that forms another part of theobject, as is conventionally done with meta-data). The watermark caninclude text saying “sunset” or the like. More compact informationrepresentations can alternatively be employed (e.g. coded references).Still further, the watermark can include (or consist entirely of) aUnique ID (UID) that serves as an index (key) into a network-connectedremote database containing the meta data descriptors. By sucharrangements, web crawlers and the like can extract and index themeta-data descriptor tags, allowing searches to be conducted based onsemantic descriptions of the file contents, rather than just by filename.

Existing watermarks commonly embed information serving to communicatecopyright information. Some systems embed text identifying the copyrightholder. Others embed a UID which is used as an index into a databasewhere the name of the copyright owner, and associated information, isstored.

Looking ahead, watermarks should serve more than as silent copyrightnotices. One option is to use watermarks to embed “intelligence” incontent. One form of intelligence is knowing its “home.” “Home” can bethe URL of a site with which the content is associated. A photograph ofa car, for example, can be watermarked with data identifying the website of an auto-dealer that published the image. Wherever the imagegoes, it serves as a link back to the original disseminator. The sametechnique can be applied to corporate logos. Wherever they are copied onthe internet, a suitably-equipped browser or the like can decode thedata and link back to the corporation's home page. (Decoding may beeffected by positioning the cursor over the logo and pressing theright-mouse button, which opens a window of options—one of which isDecode Watermark.)

To reduce the data load of the watermark, the intelligence need not bewholly encoded in the content's watermark. Instead, the watermark canagain provide a UID—this time identifying a remote database record wherethe URL of the car dealer, etc., can be retrieved. In this manner,images and the like become marketing agents—linking consumers withvendors (with some visual salesmanship thrown in). In contrast to thecopyright paradigm, in which dissemination of imagery was an evil soughtto be tracked and stopped, dissemination of the imagery can now betreated as a selling opportunity. A watermarked image becomes a portalto a commercial transaction.

(Using an intermediate database between a watermarked content file andits ultimate home (i.e. indirect linking) serves an important advantage:it allows the disseminator to change the “home” simply by updating arecord in the database. Thus, for example, if one company is acquired byanother, the former company's smart images can be made to point to thenew company's home web page by updating a database record. In contrast,if the old company's home URL is hard-coded (i.e. watermarked) in theobject, it may point to a URL that eventually is abandoned. In thissense, the intermediate database serves as a switchboard that couplesthe file to its current home.

The foregoing techniques are not limited to digital content files. Thesame approach is equally applicable with printed imagery, etc. A printedcatalog, for example, can include a picture illustrating a jacket.Embedded in the picture is watermarked data. This data can be extractedby a simple hand-scanner/decoder device using straightforward scanningand decoding techniques (e.g. those known to artisans in those fields).In watermark-reading applications employing hand-scanners and the like,it is important that the watermark decoder be robust to rotation of theimage, since the catalog photo will likely be scanned off-axis. Oneoption is to encode subliminal graticules (e.g. visualizationsynchronization codes) in the catalog photo so that the set of imagedata can be post-processed to restore it to proper alignment prior todecoding.

The scanner/decoder device can be coupled to a modem-equipped computer,a telephone, or any other communications device. In the former instance,the device provides URL data to the computer's web browser, linking thebrowser to the catalog vendor's order page. (The device need not includeits own watermark decoder; this task can be performed by the computer.)The vendor's order page can detail the size and color options of thejacket, inventory availability, and solicit ordering instructions(credit card number, delivery options, etc.)—as is conventionally donewith on-line merchants. Such a device connected to a telephone can dialthe catalog vendor's toll-free automated order-taking telephone number(known, e.g., from data encoded in the watermark), and identify thejacket to the order center. Voice prompts can then solicit thecustomer's choice of size, color, and delivery options, which are inputby Touch Tone instructions, or by voiced words (using known voicerecognition software at the vendor facility).

In such applications, the watermark may be conceptualized as aninvisible bar code employed in a purchase transaction. Here, aselsewhere, the watermark can serve as a seamless interface bridging theprint and digital worlds

Another way of providing content with intelligence is to use thewatermark to provide Java or ActiveX code. The code can be embedded inthe content, or can be stored remotely and linked to the content. Whenthe watermarked object is activated, the code can be executed (eitherautomatically, or at the option of the user). This code can performvirtually any function. One is to “phone home”—initiating a browser andlinking to the object's home. The object can then relay any manner ofdata to its home. This data can specify some attribute of the data, orits use. The code can also prevent accessing the underlying contentuntil permission is received. An example is a digital movie that, whendouble-clicked, automatically executes a watermark-embedded Java appletwhich links through a browser to the movie's distributor. The user isthen prompted to input a credit card number. After the number has beenverified and a charge made, the applet releases the content of the fileto the computer's viewer for viewing of the movie. Support for theseoperations is desirably provided via the computer's operating system, orplug-in software.

Such arrangements can also be used to collect user-provided demographicinformation when smart image content is accessed by the consumer of thecontent. The demographic information can be written to a remote databaseand can be used for market research, customization of information aboutthe content provided to the consumer, sales opportunities, advertising,etc.

In audio and video and the like, watermarks can serve to convey relatedinformation, such as links to WWW fan sites, actor biographies,advertising for marketing tie-ins (T-shirts, CDs, concert tickets). Insuch applications, it is desirable (but not necessary) to display on theuser interface (e.g. screen) a small logo to signal the presence ofadditional information. When the consumer selects the logo via someselection device (mouse, remote control button, etc.), the informationis revealed to the consumer, who can then interact with it.

Much has been written (and patented) on the topic of asset rightsmanagement. Sample patent documents include U.S. Pat. Nos. 5,715,403,5,638,443, 5,634,012, 5,629,980. Again, much of the technical work ismemorialized in journal articles, which can be identified by searchingfor relevant company names and trademarks such as IBM's Cryptolopesystem, Portland Software's ZipLock system, the Rights Exchange serviceby Softbank Net Solutions, and the DigiBox system from InterTrustTechnologies.

An exemplary asset management system makes content available (e.g. froma web server, or on a new computer's hard disk) in encrypted form.Associated with the encrypted content is data identifying the content(e.g. a preview) and data specifying various rights associated with thecontent. If a user wants to make fuller use of the content, the userprovides a charge authorization (e.g. a credit card) to the distributor,who then provides a decryption key, allowing access to the content.(Such systems are often realized using object-based technology. In suchsystems, the content is commonly said to be distributed in a “securecontainer.”)

Desirably, the content should be marked (personalized/serialized) sothat any illicit use of the content (after decryption) can be tracked.This marking can be performed with watermarking, which assures that themark travels with the content wherever—and in whatever form—it may go.The watermarking can be effected by the distributor—prior todissemination of the encrypted object—such as by encoding a UID that isassociated in a database with that particular container. When accessrights are granted to that container, the database record can be updatedto reflect the purchaser, the purchase date, the rights granted, etc. Analternative is to include a watermark encoder in the software tool usedto access (e.g. decrypt) the content. Such an encoder can embedwatermark data in the content as it is released from the securecontainer, before it is provided to the user. The embedded data caninclude a UID, as described above. This UID can be assigned by thedistributor prior to disseminating the container. Alternatively, the UIDcan be a data string not known or created until access rights have beengranted. In addition to the UID, the watermark can include other datanot known to the distributor, e.g. information specific to the time(s)and manner(s) of accessing the content.

In other systems, access rights systems can be realized with watermarkswithout containers etc. Full resolution images, for example, can befreely available on the web. If a user wishes to incorporate the imageryinto a web page or a magazine, the user can interrogate the imagery asto its terms and conditions of use. This may entail linking to a website specified by the embedded watermark (directly, or through anintermediate database), which specifies the desired information. Theuser can then arrange the necessary payment, and use the image knowingthat the necessary rights have been secured.

As noted, digital watermarks can also be realized using conventional(e.g. paper) watermarking technologies. Known techniques forwatermarking media (e.g. paper, plastic, polymer) are disclosed in U.S.Pat. Nos. 5,536,468, 5,275,870, 4,760,239, 4,256,652, 4,370,200, and3,985,927 and can be adapted to display of a visual watermark instead ofa logo or the like. Note that some forms of traditional watermarks whichare designed to be viewed with transmissive light can also show up aslow level signals in reflective light, as is typically used in scanners.Transmissive illumination detection systems can also be employed todetect such watermarks, using optoelectronic traditional-watermarkdetection technologies known in the art.

As also noted, digital watermarks can be realized as part of opticalholograms. Known techniques for producing and securely mountingholograms are disclosed in U.S. Pat. Nos. 5,319,475, 5,694,229,5,492,370, 5,483,363, 5,658,411 and 5,310,222. To watermark a hologram,the watermark can be represented in the image or data model from whichthe holographic diffraction grating is produced. In one embodiment, thehologram is produced as before, and displays an object or symbol. Thewatermark markings appear in the background of the image so that theycan be detected from all viewing angles. In this context, it is notcritical that the watermark representation be essentially imperceptibleto the viewer. If desired, a fairly visible noise-like pattern can beused without impairing the use to which the hologram is put.

Digital watermarks can also be employed in conjunction with labels andtags. In addition to conventional label/tag printing processes, othertechniques—tailored to security—can also be employed. Known techniquesuseful in producing security labels/tags are disclosed in U.S. Pat. Nos.5,665,194, 5,732,979, 5,651,615, and 4,268,983. The imperceptibility ofwatermarked data, and the ease of machine decoding, are some of thebenefits associated with watermarked tags/labels. Additionally, the costis far less than many related technologies (e.g. holograms). Watermarksin this application can be used to authenticate the originality of aproduct label, either to the merchant or to the consumer of theassociated product, using a simple scanner device, thereby reducing therate of counterfeit product sales.

Recent advances in color printing technology have greatly increased thelevel of casual counterfeiting. High quality scanners are now readilyavailable to many computer users, with 300 dpi scanners available forunder $100, and 600 dpi scanners available for marginally more.Similarly, photographic quality color ink-jet printers are commonlyavailable from Hewlett-Packard Co., Epson, etc. for under $300.

Watermarks in banknotes and other security documents (passports, stockcertificates, checks, etc.—all collectively referred to as banknotesherein) offer great promise to reduce such counterfeiting, as discussedmore fully below. Additionally, watermarks provide a high-confidencetechnique for banknote authentication. One product enabled by thisincreased confidence is automatic teller machines that accept, as wellas dispense, cash. The machine is provided with known optical scanningtechnology to produce digital data corresponding to the face(s) of thebill. This image set is then analyzed to extract the watermark data. Inwatermarking technologies that require knowledge of a code signal fordecoding (e.g. noise modulation signal, crypto key, spreading signal,etc.), a bill may be watermarked in accordance with several such codes.Some of these codes are public—permitting their reading by conventionalmachines. Others are private, and are reserved for use by governmentagencies and the like. (C.f. public and private codes in the presentassignee's issued patents.)

Banknotes presently include certain markings which can be used as an aidin note authentication. Well known visible structures are added tobanknotes to facilitate visual authentication and machine detection. Anexample is the seal of the issuing bank. Others are geometricalmarkings. Desirably, a note is examined by an integrated detectionsystem, for both such visible structures as well as the presentwatermark-embedded data, to determine authenticity.

The visible structures can be sensed using known pattern recognitiontechniques. Examples of such techniques are disclosed in U.S. Pat. Nos.5,321,773, 5,390,259, 5,533,144, 5,539,841, 5,583,614, 5,633,952,4,723,149 and 5,424,807 and laid-open foreign application EP 766,449.The embedded watermark data can be recovered using the scanning/analysistechniques disclosed in the cited patents and publications.

To reduce counterfeiting, it is desirable that document-reproducingtechnologies recognize banknotes and refuse to reproduce same. Aphotocopier, for example, can sense the presence of either a visiblestructure *or* embedded banknote watermark data, and disable copying ifeither is present. Scanners and printers can be equipped with a similarcapability—analyzing the data scanned or to be printed for either ofthese banknote hallmarks. If either is detected, the software (orhardware) disables further operation.

The watermark detection criteria provides an important advantage nototherwise available. An original bill can be doctored (e.g. bywhite-out, scissors, or less crude techniques) to remove/obliterate thevisible structures. Such a document can then be freely copied on eithera visible structure-sensing photocopier or scanner/printer installation.The removed visible structure can then be added in via a secondprinting/photocopying operation. If the printer is not equipped withbanknote-disabling capabilities, image-editing tools can be used toinsert visible structures back into image data sets scanned from suchdoctored bills, and the complete bill freely printed. By additionallyincluding embedded watermark data in the banknote, and sensing same,such ruses will not succeed.

(A similar ruse is to scan a banknote image on a non-banknote-sensingscanner. The resulting image set can then be edited by conventionalimage editing tools to remove/obliterate the visible structures. Such adata set can then be printed—even on a printer/photocopier that examinessuch data for the presence of visible structures. Again, the missingvisible structures can be inserted by a subsequent printing/photocopyingoperation.)

Desirably, the visible structure detector and the watermark detector areintegrated together as a single hardware and/or software tool. Thisarrangement provides various economies, e.g., in interfacing with thescanner, manipulating pixel data sets for pattern recognition andwatermark extraction, electronically re-registering the image tofacilitate pattern recognition/watermark extraction, issuing controlsignals (e.g. disabling) signals to the photocopier/scanner, etc.

A related principle is to insert an imperceptible watermark having a UIDinto all documents printed with a printer, scanned with a scanner, orreproduced by a photocopier. The UID is associated with the particularprinter/photocopier/scanner in a registry database maintained by theproducts' manufacturers. The manufacturer can also enter in thisdatabase the name of the distributor to whom the product was initiallyshipped. Still further, the owner's name and address can be added to thedatabase when the machine is registered for warranty service. While notpreventing use of such machines in counterfeiting, the embedded UIDfacilitates identifying the machine that generated a counterfeitbanknote. (This is an application in which a private watermark mightbest be used.)

While the foregoing applications disabled potential counterfeitingoperations upon the detection of *either* a visible structure orwatermarked data, in other applications, both criteria must be metbefore a banknote is recognized as genuine. Such applications typicallyinvolve the receipt or acceptance of banknotes, e.g. by ATMs asdiscussed above.

The foregoing principles (employing just watermark data, or inconjunction with visible indicia) can likewise be used to preventcounterfeiting of tags and labels (e.g. the fake labels and tagscommonly used in pirating Levis brand jeans, Microsoft software, etc.)

The reader may first assume that banknote watermarking is effected byslight alterations to the ink color/density/distribution, etc. on thepaper. This is one approach. Another is to watermark the underlyingmedium (whether paper, polymer, etc.) with a watermark. This can be doneby changing the microtopology of the medium (a la mini-Braille) tomanifest the watermark data. Another option is to employ a laminate onor within the banknote, where the laminate has the watermarkingmanifested thereon/therein. The laminate can be textured (as above), orits optical transmissivity can vary in accordance with a noise-likepattern that is the watermark, or a chemical property can similarlyvary.

Another option is to print at least part of a watermark usingphotoluminescent ink. This allows, e.g., a merchant presented with abanknote, to quickly verify the presence of *some* watermark-likeindicia in/on the bill even without resort to a scanner and computeranalysis (e.g. by examining under a black light). Such photoluminescentink can also print human-readable indicia on the bill, such as thedenomination of a banknote. (Since ink-jet printers and other commonmass-printing technologies employ cyan/magenta/yellow/black to formcolors, they can produce only a limited spectrum of colors.Photoluminescent colors are outside their capabilities. Fluorescentcolors—such as the yellow, pink and green dyes used in highlightingmarkers—can similarly be used and have the advantage of being visiblewithout a black light.)

An improvement to existing encoding techniques is to add an iterativeassessment of the robustness of the mark, with a correspondingadjustment in a re-watermarking operation. Especially when encodingmultiple bit watermarks, the characteristics of the underlying contentmay result in some bits being more robustly (e.g. strongly) encoded thanothers. In an illustrative technique employing this improvement, awatermark is first embedded in an object. Next, a trial decodingoperation is performed. A confidence measure (e.g. signal-to-noiseratio) associated with each bit detected in the decoding operation isthen assessed. The bits that appear weakly encoded are identified, andcorresponding changes are made to the watermarking parameters to bringup the relative strengths of these bits. The object is then watermarkedanew, with the changed parameters. This process can be repeated, asneeded, until all of the bits comprising the encoded data areapproximately equally detectable from the encoded object, or meet somepredetermined signal-to-noise ratio threshold.

The foregoing applications, and others, can generally benefit bymultiple watermarks. For example, an object (physical or data) can bemarked once in the spatial domain, and a second time in the spatialfrequency domain. (It should be understood that any change in one domainhas repercussions in the other. Here we reference the domain in whichthe change is directly effected.)

Another option is to mark an object with watermarks of two differentlevels of robustness, or strength. The more robust watermark withstandsvarious types of corruption, and is detectable in the object even aftermultiple generations of intervening distortion. The less robustwatermark can be made frail enough to fail with the first distortion ofthe object. In a banknote, for example, the less robust watermark servesas an authentication mark. Any scanning and reprinting operation willcause it to become unreadable. Both the robust and the frail watermarksshould be present in an authentic banknote; only the former watermarkwill be present in a counterfeit.

Still another form of multiple-watermarking is with content that iscompressed. The content can be watermarked once (or more) in anuncompressed state. Then, after compression, a further watermark (orwatermarks) can be applied.

Still another advantage from multiple watermarks is protection againstsleuthing. If one of the watermarks is found and cracked, the otherwatermark(s) will still be present and serve to identify the object.

The foregoing discussion has addressed various technological fixes tomany different problems. Exemplary solutions have been detailed above.Others will be apparent to the artisan by applying common knowledge toextrapolate from the solutions provided above.

For example, the technology and solutions disclosed herein have made useof elements and techniques known from the cited references. Otherelements and techniques from the cited references can similarly becombined to yield further implementations within the scope of thepresent invention. Thus, for example, holograms with watermark data canbe employed in banknotes, single-bit watermarking can commonly besubstituted for multi-bit watermarking, technology described as usingimperceptible watermarks can alternatively be practiced using visiblewatermarks (glyphs, etc.), techniques described as applied to images canlikewise be applied to video and audio, local scaling of watermarkenergy can be provided to enhance watermark signal-to-noise ratiowithout increasing human perceptibility, various filtering operationscan be employed to serve the functions explained in the prior art,watermarks can include subliminal graticules to aid in imagere-registration, encoding may proceed at the granularity of a singlepixel (or DCT coefficient), or may similarly treat adjoining groups ofpixels (or DCT coefficients), the encoding can be optimized to withstandexpected forms of content corruption. Etc., etc., etc. Thus, theexemplary embodiments are only selected samples of the solutionsavailable by combining the teachings referenced above. The othersolutions necessarily are not exhaustively described herein, but arefairly within the understanding of an artisan given the foregoingdisclosure and familiarity with the cited art.

1. A method comprising: obtaining first media comprising imagery or audio; obtaining second media comprising imagery or audio; aggregating the first media and the second media to yield a unit; and using a programmed electronic processor, examining the unit for the presence of a digital watermark.
 2. The method of claim 1 in which the digital watermark is manifested in the first media or the second media as changes to transform domain values.
 3. The method of claim 1 in which the imagery comprises video.
 4. The method of claim 1 in which the first media or the second media comprises audio.
 5. The method of claim 1 in which the first media comprises a first image and the second media comprises a second image, and said act of aggregating aggregates image tiles of the first image and the second image.
 6. The method of claim 1 wherein a web crawler performs the method.
 7. A computer readable medium comprising instructions stored thereon to cause an electronic processor to perform the method of claim
 1. 8. A computer readable medium comprising instructions stored thereon to cause an electronic processor to perform the method of claim
 6. 9. A computing device programmed to perform the method of claim
 1. 10. A programmed computing device comprising instructions stored in memory, said instructions cause said programmed computing device to perform the method of claim
 1. 11. A method comprising: upon encountering imagery or audio content, performing a screening operation on the imagery or audio content; based at least in part on the result of the screening operation, determining whether to derive or extract identifying data from the imagery or audio content, said act of determining uses a programmed electronic processor; providing the identifying data for storage in an electronic data record.
 12. The method of claim 11 in which said screening operation comprises a transform domain analysis of the imagery or audio content.
 13. The method of claim 11 in which the imagery or audio content is encountered by a web crawler.
 14. The method of claim 11 in which encountered imagery or audio content comprises digital watermarking therein, the digital watermarking conveying the identifying data, and in which the identifying data is derived or extracted with digital watermark decoding.
 15. A computer readable medium comprising instructions stored thereon to cause an electronic processor to perform the method of claim
 11. 16. A computing device programmed to perform the method of claim
 11. 17. A programmed computing device comprising instructions stored in memory, said instructions cause said programmed computing device to perform the method of claim
 11. 18. The method of claim 11 in which the screening operation identifies content likely to include digital watermarking.
 19. A method comprising: using a programmed electronic processor, extracting identifying data from data representing imagery or from data representing audio; automatically dialing a telephone number; providing the identifying data for telephonic communication to a device hosting the telephone number; receiving signals representing a human voice command; and providing signals for telephonic communication to the device hosting the telephone number.
 20. The method of claim 19 in which the identifying data is extracted from the data representing imagery or from the data representing audio using digital watermark decoding.
 21. A computer readable medium comprising instructions stored thereon to cause an electronic processor to perform the method of claim
 19. 22. A computing device programmed to perform the method of claim
 19. 23. A programmed computing device comprising instructions stored in memory, said instructions cause said programmed computing device to perform the method of claim
 19. 24. An apparatus comprising: one or more electronic processors programmed for: upon encountering imagery or audio content, performing a screening operation on the imagery or audio content; based at least in part on the result of the screening operation, determining whether to derive or extract identifying data from the imagery or audio content; and providing the identifying data for storage in an electronic data record.
 25. The apparatus of claim 24 further comprising the electronic data record.
 26. The apparatus of claim 24 in which the screening operation comprises a transform domain analysis of the imagery or audio content.
 27. The apparatus of claim 24 in which the apparatus comprises a web crawler.
 28. The apparatus of claim 24 in which encountered imagery or audio content comprises digital watermarking therein, the digital watermarking conveying the identifying data, and in which the identifying data is derived or extracted with digital watermark decoding.
 29. The apparatus of claim 24 in which the screening operation identifies content likely to include digital watermarking.
 30. The apparatus of claim 24 in which said one or more electronic processors are operating to perform at least one of the functions recited therein.
 31. The apparatus of claim 26 in which said one or more electronic processors are operating to perform at least one of the functions recited therein. 